Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Edit the manifest when pushing uncompressed data from c/storage #2273

Merged
merged 5 commits into from
Feb 7, 2024
Merged

Edit the manifest when pushing uncompressed data from c/storage #2273

merged 5 commits into from
Feb 7, 2024

Conversation

mtrmac
Copy link
Collaborator

@mtrmac mtrmac commented Jan 31, 2024 •
edited
Loading

Fixes #2182.

At this point completely untested.

@mtrmac
Copy link
Collaborator Author

mtrmac commented Jan 31, 2024 •
edited
Loading

Downstream tests:

mtrmac added a commit to mtrmac/buildah that referenced this pull request Feb 1, 2024
@mtrmac
DO NOT MERGE: Testing containers/image#2273
1a9a63d 
Signed-off-by: Miloslav Trmač <[email protected]>
mtrmac added a commit to mtrmac/skopeo that referenced this pull request Feb 2, 2024
@mtrmac
DO NOT MERGE: Testing containers/image#2273
d1627d1 
Signed-off-by: Miloslav Trmač <[email protected]>
mtrmac added a commit to mtrmac/skopeo that referenced this pull request Feb 2, 2024
@mtrmac
DO NOT MERGE: Testing containers/image#2273
188e85d 
Signed-off-by: Miloslav Trmač <[email protected]>
mtrmac added a commit to mtrmac/skopeo that referenced this pull request Feb 3, 2024
@mtrmac
DO NOT MERGE: Testing containers/image#2273
f2c44dc 
Signed-off-by: Miloslav Trmač <[email protected]>
mtrmac added a commit to mtrmac/skopeo that referenced this pull request Feb 3, 2024
@mtrmac
DO NOT MERGE: Testing containers/image#2273
2fda8e1 
Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
Copy link
Collaborator Author

mtrmac commented Feb 5, 2024

I have manually tested that in Podman with containers/podman#21461 , this fixes the original reproducer of #2182.

Also, Skopeo’s tests demonstrate the same behavior — and rely on it, which is why they are failing here. containers/skopeo#2213 will update Skopeo.

@mtrmac mtrmac force-pushed the storage-source-no-compression-change-edits branch from b31f2b0 to f02a40f Compare February 5, 2024 21:43
mtrmac added a commit to mtrmac/skopeo that referenced this pull request Feb 5, 2024
@mtrmac
DO NOT MERGE: Testing containers/image#2273
3abb69a 
Signed-off-by: Miloslav Trmač <[email protected]>
mtrmac added a commit to mtrmac/buildah that referenced this pull request Feb 5, 2024
@mtrmac
DO NOT MERGE: Testing containers/image#2273
b1d6b3f 
Signed-off-by: Miloslav Trmač <[email protected]>
mtrmac added a commit to mtrmac/libpod that referenced this pull request Feb 5, 2024
@mtrmac
DO NOT MERGE: Testing containers/image#2273
9d656cc 
Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
Copy link
Collaborator Author

mtrmac commented Feb 6, 2024

Ready for review. Downstream tests are mostly passing.

The Skopeo test failures here expose the previous incorrect behavior; containers/skopeo#2213 contains test updates, and passes.

Cc: @giuseppe

@mtrmac mtrmac marked this pull request as ready for review February 6, 2024 18:17
mtrmac added a commit to mtrmac/buildah that referenced this pull request Feb 6, 2024
@mtrmac
DO NOT MERGE: Testing containers/image#2273
6a8e776 
Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac mtrmac changed the title WIP: Edit the manifest when pushing uncompressed data from c/storage Edit the manifest when pushing uncompressed data from c/storage Feb 6, 2024
giuseppe
giuseppe approved these changes Feb 7, 2024
View reviewed changes
Copy link
Member

@giuseppe giuseppe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

mtrmac added a commit to mtrmac/buildah that referenced this pull request Feb 7, 2024
@mtrmac
DO NOT MERGE: Testing containers/image#2273
feaf725 
Signed-off-by: Miloslav Trmač <[email protected]>
mtrmac added a commit to mtrmac/libpod that referenced this pull request Feb 7, 2024
@mtrmac
DO NOT MERGE: Testing containers/image#2273
58fdaaf 
Signed-off-by: Miloslav Trmač <[email protected]>
@rhatdan
Copy link
Member

rhatdan commented Feb 7, 2024

Tests are failing?

@mtrmac
Copy link
Collaborator Author

mtrmac commented Feb 7, 2024

Yes, as documented in detail above.

@rhatdan
Copy link
Member

rhatdan commented Feb 7, 2024

LGTM

@mtrmac
Reword the documentation a bit to suggest the edits are idempotent
27813fb 
In particular, it is OK to use CompressionOperation: Decompress
on already-decompressed layers.

TestUpdatedMIMEType already ensures this works correctly.

Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
Rename bpCompressionStepData.operation to uploadedOperation
230e72f 
Make it clearer that it is intended for manifest edits, and is
not indicative of what we actually did.

Should not change behavior.

Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
Add a new bpCompressionStepData.operation
61312cf 
... describing in more granularity what we are doing.

Use it in recordValidatedDigestData, adding a case where
we can record that a layer is uncompressed when we determine that
while copying it without changes.

Adding this separate .operation field will also allow us to set
.uploadedOperation to Decompress in the bpcOpPreserveUncompressed
case, without changing what recordValidatedDigestData does.

Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
Ensure uncompressed layers from c/storage are recorded as uncompresse…
772f391 
…d on upload

When copying an uncompressed layer without modification, arrange
types.BlobInfo.CompressionOperation to be set to types.Decompress,
so that we edit the manifest to use an uncompressed MIME type
(and not, e.g. a zstd:chunked MIME type which might not be representable
in the destination at all).

This is all still gated by the
> 	if srcInfosUpdated || layerDigestsDiffer(srcInfos, destInfos) {
condition in copyLayers, so we don't do these manifest updates
frivolously.

Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
Ensure uncompressed layers from c/storage are recorded as uncompresse…
278b324 
…d on reuse

Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac mtrmac force-pushed the storage-source-no-compression-change-edits branch from f02a40f to 278b324 Compare February 7, 2024 21:44
@mtrmac mtrmac merged commit 93b4b55 into containers:main Feb 7, 2024
7 of 10 checks passed
@mtrmac mtrmac deleted the storage-source-no-compression-change-edits branch February 7, 2024 23:14
mtrmac added a commit to mtrmac/skopeo that referenced this pull request Feb 7, 2024
@mtrmac
Update c/image after containers/image#2273
c3e2b20 
This is necessary so that c/image tests can pass.

Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac mtrmac mentioned this pull request Feb 7, 2024
Merged
mtrmac added a commit to containers/skopeo that referenced this pull request Feb 7, 2024
@mtrmac
Merge pull request #2213 from mtrmac/c-image-decompress-edits
a1b5b96 
Update after containers/image#2273
atmosphere-ci bot pushed a commit to vexxhost/atmosphere that referenced this pull request Apr 21, 2024
@renovate
chore(deps): update module github.com/containers/image/v5 to v5.30.0 (#…
ec06757 
…1114)

This PR contains the following updates:



Package
Change
Age
Adoption
Passing
Confidence




github.com/containers/image/v5
v5.29.2 -> v5.30.0








WarningSome dependencies could not be looked up. Check the warning logs for more information.


Release Notes

containers/image (github.com/containers/image/v5)
v5.30.0
Compare Source
What's Changed
A fair number of improvements when working with zstd and zstd:chunked-compressed images.
Note that make install now installs policy.json and registries.d/default.yaml.

Refuse compression to zstd when using schema1 by @​mtrmac in containers/image#2196
Don't expose local account details in oci-archive tar files by @​mtrmac in containers/image#2202
Trigger a conversion to OCI when compressing to Zstd by @​mtrmac in containers/image#2204
Add buildtags to avoid fulcio and rekor dependencies by @​siretart in containers/image#2180
copy: do not fail if digest mismatches by @​giuseppe in containers/image#1980
Moving policy.json and default.yaml from containers/skopeo by @​rahilarious in containers/image#2215
Embrace codespell: config, workflow (to alert when new typos added) and get typos fixed by @​yarikoptic in containers/image#2214
Fix raspberry pi zero cpu variant recognition by @​lstolcman in containers/image#2086
storage: validate images converted to zstd:chunked by @​giuseppe in containers/image#2243
Make blob reuse choices manifest-format-sensitive, and allow conversions when writing to format-agnostic transports by @​mtrmac in containers/image#2213
Edit the manifest when pushing uncompressed data from c/storage by @​mtrmac in containers/image#2273
Random storage-related cleanups by @​mtrmac in containers/image#2287
Improve storage transport documentation, primarily about locking by @​mtrmac in containers/image#2291
Fix c/storage destination with partial pulls by @​mtrmac in containers/image#2288
Fix manifest updates when we match a layer by TOC digest by @​mtrmac in containers/image#2294
Cleanly fail when trying to obtain a DiffID of a non-OCI image by @​mtrmac in containers/image#2295
Beautify TOC-related parts of storageImageSource by @​mtrmac in containers/image#2296
storage: use the new ApplyStagedLayer interface by @​giuseppe in containers/image#2301
Also annotate image instances using zstd:chunked as using zstd by @​mtrmac in containers/image#2302
Support editing ArtifactType, preserve it in lists by @​nalind in containers/image#2304
Provide data to correctly report throughput on partial pulls by @​mtrmac in containers/image#2308
Add validation error to digesting reader by @​saschagrunert in containers/image#2312
Fix handling of errors when fetching layers by URLs by @​mtrmac in containers/image#2310
Improve handling of zstd vs. zstd:chunked matching by @​mtrmac in containers/image#2317

New Contributors

@​rahilarious made their first contribution in containers/image#2215
@​yarikoptic made their first contribution in containers/image#2214
@​lstolcman made their first contribution in containers/image#2086
@​bainsy88 made their first contribution in containers/image#2260

Full Changelog: containers/[email protected]


Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.


 If you want to rebase/retry this PR, check this box


This PR has been generated by Mend Renovate. View repository job log here.
doanac pushed a commit to lmp-mirrors/meta-virtualization that referenced this pull request Jul 16, 2024
@zeddii
skopeo: update to v1.15.2
13a0ab6 
Bumping skopeo to version v1.15.1-8-g80626759, which comprises the following commits:

    0002d9ca [release-1.15] Bump to v1.15.2
    260f444b [release-1.15] Fixes an interoperability issue while listing tags
    cda43af4 [skip-ci] Packit: enable c10s downstream sync
    7e81106f Refer to registry.k8s.io instead of k8s.gcr.io
    e2ea4269 [release-1.15] Bump to v1.15.1
    5267baab [release-1.15] Freeze the fedora-minimal image reference
    cd5f7090 [release-1.15] CVE-2024-3727
    99e3c448 Bump to v1.15.0
    6d6d5b7b chore(deps): update module google.golang.org/protobuf to v1.33.0 [security]
    78ddfd9d fix(deps): update module github.com/containers/ocicrypt to v1.1.10
    e46c71c3 chore(deps): update module gopkg.in/go-jose/go-jose.v2 to v2.6.3 [security]
    549fc862 chore(deps): update module github.com/go-jose/go-jose/v3 to v3.0.3 [security]
    cca855ac [CI:DOCS] Migrate skopeo container image
    2ffc4ec3 fix(deps): update module github.com/containers/common to v0.58.0
    086701bd fix(deps): update module github.com/containers/image/v5 to v5.30.0
    96443048 fix(deps): update module github.com/containers/storage to v1.53.0
    a80f634f fix(deps): update module golang.org/x/term to v0.18.0
    4d80bf8c fix(deps): update github.com/containers/image/v5 digest to faa4f4f
    2db09313 Move a go.mod dependency entry to the right section
    95f6eac4 Avoid a warning by gopls / VSCode
    dbf465d6 fix typo
    55abe69d update docs based on review
    439dd5f4 docs: make sure that CGO_ENABLED='1'
    f0b07848 [CI:DOCS] Update dependency golangci/golangci-lint to v1.56.2
    8b6774b4 fix(deps): update module github.com/opencontainers/image-spec to v1.1.0
    e294a380 [CI:DOCS] Update dependency golangci/golangci-lint to v1.56.1
    c3e2b202 Update c/image after containers/image#2273
    8f2a7c65 Also change the MIME type in manifest when decompressing a layer
    812a02af Replace decompress-dirs.sh with Go code
    09f282e4 Split decompressDir from runDecompressDirs
    9fd98112 [CI:DOCS] Update dependency golangci/golangci-lint to v1.56.0
    122bf968 fix(deps): update module github.com/containers/common to v0.57.4
    149fec0a fix(deps): update golang.org/x/exp digest to 1b97071
    d77f7153 fix(deps): update module github.com/containers/common to v0.57.3
    c941d813 Update github.com/containers/image/v5 to post-v5.29.2 main
    b412ed6e Run codespell on code
    58ff9fdb fix(deps): update module github.com/containers/storage to v1.52.0
    177d4adb add kubasobon:semver changes
    4baf4fe9 fix(deps): update module github.com/opencontainers/image-spec to v1.1.0-rc6
    e32dfefb Bump to v1.15.0-dev

Signed-off-by: Bruce Ashfield <[email protected]>
daregit pushed a commit to daregit/yocto-combined that referenced this pull request Aug 1, 2024
@zeddii
src/meta-virtualization:skopeo: update to v1.15.2
bb77cd6 
Bumping skopeo to version v1.15.1-8-g80626759, which comprises the following commits:

    0002d9ca [release-1.15] Bump to v1.15.2
    260f444b [release-1.15] Fixes an interoperability issue while listing tags
    cda43af4 [skip-ci] Packit: enable c10s downstream sync
    7e81106f Refer to registry.k8s.io instead of k8s.gcr.io
    e2ea4269 [release-1.15] Bump to v1.15.1
    5267baab [release-1.15] Freeze the fedora-minimal image reference
    cd5f7090 [release-1.15] CVE-2024-3727
    99e3c448 Bump to v1.15.0
    6d6d5b7b chore(deps): update module google.golang.org/protobuf to v1.33.0 [security]
    78ddfd9d fix(deps): update module github.com/containers/ocicrypt to v1.1.10
    e46c71c3 chore(deps): update module gopkg.in/go-jose/go-jose.v2 to v2.6.3 [security]
    549fc862 chore(deps): update module github.com/go-jose/go-jose/v3 to v3.0.3 [security]
    cca855ac [CI:DOCS] Migrate skopeo container image
    2ffc4ec3 fix(deps): update module github.com/containers/common to v0.58.0
    086701bd fix(deps): update module github.com/containers/image/v5 to v5.30.0
    96443048 fix(deps): update module github.com/containers/storage to v1.53.0
    a80f634f fix(deps): update module golang.org/x/term to v0.18.0
    4d80bf8c fix(deps): update github.com/containers/image/v5 digest to faa4f4f
    2db09313 Move a go.mod dependency entry to the right section
    95f6eac4 Avoid a warning by gopls / VSCode
    dbf465d6 fix typo
    55abe69d update docs based on review
    439dd5f4 docs: make sure that CGO_ENABLED='1'
    f0b07848 [CI:DOCS] Update dependency golangci/golangci-lint to v1.56.2
    8b6774b4 fix(deps): update module github.com/opencontainers/image-spec to v1.1.0
    e294a380 [CI:DOCS] Update dependency golangci/golangci-lint to v1.56.1
    c3e2b202 Update c/image after containers/image#2273
    8f2a7c65 Also change the MIME type in manifest when decompressing a layer
    812a02af Replace decompress-dirs.sh with Go code
    09f282e4 Split decompressDir from runDecompressDirs
    9fd98112 [CI:DOCS] Update dependency golangci/golangci-lint to v1.56.0
    122bf968 fix(deps): update module github.com/containers/common to v0.57.4
    149fec0a fix(deps): update golang.org/x/exp digest to 1b97071
    d77f7153 fix(deps): update module github.com/containers/common to v0.57.3
    c941d813 Update github.com/containers/image/v5 to post-v5.29.2 main
    b412ed6e Run codespell on code
    58ff9fdb fix(deps): update module github.com/containers/storage to v1.52.0
    177d4adb add kubasobon:semver changes
    4baf4fe9 fix(deps): update module github.com/opencontainers/image-spec to v1.1.0-rc6
    e32dfefb Bump to v1.15.0-dev

Signed-off-by: Bruce Ashfield <bruce.ashfieldgmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@giuseppe giuseppe giuseppe approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@mtrmac @rhatdan @giuseppe